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ABSTRACT 

This document reports on a workshop held January 4-5, 1996. 
Participants of this conference represented the areas of science, 
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"Introduction"; (2) "Evaluation Questions Conference"; and (3) "Next Steps." 
Appendices include: (1) "Conference Participants"; (2) "Invitees Unable to 
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Executive Summary 

The Evaluation Questions Conference, held in Madison, WI January 4-5, 1996, was attended by 26 
participants representing a wide range of involvement in science, mathematics, engineering, and 
technology (SMET) education and practice. The purpose of the conference was to develop a set of 
questions for and about evaluations of systemic education reform (SER). 

Four discussion papers, authored by conference participants, were distributed prior to the conference. 
Each paper enumerated a list of questions for and about an evaluation of SER centering on a particular 
area of systemic reform. The areas covered were student outcomes in science and mathematics, system 
alignment and system change, and the community college. 

Following brief presentations on the four discussion papers, conference participants were divided into 
four groups for the task of further defining and refining questions for and about evaluations of SER. The 
four small working groups of 7-8 participants met for three extended periods over the two days of the 
meeting. Each group approached the task differently, but each group produced a useful, refined set of 
questions by the end of meeting. Furthermore, the groups variously produced useful definitions of 
systemic reform, raised issues about the design of evaluations of SER, and identified areas still in need of 
evaluation questions. 

The questions produced by the meeting can be roughly classified in the following categories: 

A. Questions about an evaluation of SER 

1. about evaluation design 

2. about uses of evaluation 

3. about who is involved in evaluation 

B. Questions for an evaluation SER 

1. about the nature of the reform effort — its durability, flexibility, coherence, and 
systemic nature 

2. about involvement in the design and implementation of the reform 

3. about incentives for participation in the reform 

4. about the vision of the reform 

5. about resources for the reform 

6. about the design of communication within the reform 

7. about the patterns of communication within the reform 

8. about changes in expectations 

9. about student performance outcomes 

10. about equity 

1 1 . about long-term outcomes (e.g. employment) 

12. about changes in classroom processes 

13. about professional development of teachers 

A number of next steps in the process of designing an evaluation of SER were advanced. Refinement of 
the questions into a hierarchical structure was suggested. Several different recommendations were made 
about how the questions could be used to begin thinking about evaluation designs and related issues. 

Also, some participants advanced ideas about how hypothetical or experimental evaluations could be 
conducted to examine the questions in a realistic context. 

iv 
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Introduction 



A large effort has been mounted by National Science Foundation and others to support systemic 
education reform at the state, regional, and district levels. The size and complexity of systemic reform 
creates an enormous challenge for developing evaluations that truly represent what are outcomes 
attributable to systemic education reform and can produce useful information for making decisions for 
advancing reform. A number of evaluation efforts currently in place are designed to respond to particular 
needs, i.e. external and internal evaluations of state systemic initiatives and evaluations of the total 
systemic initiative program. 

The Evaluation Strategies Project of the National Institute for Science Education is devoted towards 
studying evaluation of systemic education reform. The ES Project, composed of 13 members 
representing a range of disciplines and roles, is one of six projects of the National Institute for Science 
Education (NISE). 

One main goal for the ES Project is to design models that can be used to evaluate systemic education 
reform. As an initial step in the design process, the project set for its first year, 1995-96, the object of 
identifying what are important questions to be asked for and about an evaluation of systemic education 
reform. These questions then will be used to develop models, designs, and strategies forjudging the 
value of systemic education reform. 

Over the period from June through December, the ES Project had met three times to begin the work of 
studying the evaluation of systemic education reform. These meetings were spent establishing some 
common understanding among the project members of what is systemic reform and what is important for 
studying the evaluation of systemic reform. 

The purpose of the January 4-5, 1996, conference was to continue the process of identifying and refining 
questions critical for developing models for evaluation. Participants in the working conference received 
four discussion papers in advance of the conference. Authors of these papers were asked to write a 20 to 
30 page (double space) paper to generate discussion leading toward the identification of questions related 
to a vital area for evaluating systemic reform in science, mathematics, engineering, and technology. 

Papers were written on student outcomes in general (Burkhardt and Ridgway), student outcomes in 
science (Comfort), system alignment (Ridgway and Burkhardt), and community college issues (Jackman). 
Authors were asked to present questions, relevant to an assigned topic, for and about an evaluation of 
systemic reform and a rationale for the inclusion of each question within a coherent and informed 
framework of evaluation and systemic reform. 

Only a few people were invited to attend the conference in order to keep the number small enough for 
productive discussion. The invitees were chosen based on their discipline and the role they were 
assuming in order to have a range of stakeholders for an education system. These included classroom 
teachers, community college administrators, university professors, mathematicians, scientists, engineers, 
evaluators, and policy makers. Not all of the people could attend the conference, so not all categories 
were represented. Those who did attend were grouped into four small groups of six or seven members. 
The work of these small groups comprised the major accomplishment of the conference and is reported in 
these conference proceedings. The four discussion papers are included in the appendices. 

What is presented in this report is work in progress. The questions generated by the small groups, the 
discussion papers, and the issues raised in plenary sessions will be refined further. Questions will be 
aggregated and analyzed with respect to current literature and based on existing knowledge and 



experiences in doing evaluation of large systems. They will be used to design practical evaluation 
models. 
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